Skip to content

golive: keep the home owned after a W1 or switch that stops T3 ends (#530) - #556

Merged
scuttletoby merged 3 commits into
mainfrom
fix/golive-530-holder
Nov 20, 2026
Merged

scuttletoby merged 3 commits into
mainfrom
fix/golive-530-holder

Conversation

@scuttletoby

@scuttletoby scuttletoby commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

BASE main
Fixes #530

No longer stacked. #554 is merged; this PR now targets main. It was rebased onto the merged #554, and again onto main after main was moved on 11-19. A patch-id comparison shows that only this PR's own commits came over.

What

A W1 (and a switch) can end with T3 stopped on purpose, because the home may be half-restored. It does this when it fails after the stop, and also when it is killed. The owner lock it took after the stop belonged to the run's own process, so that lock disappeared with the run. The next SSH launch of the desktop app then found the home free and started its own server on a half-restored home.

  • The holder. Right after the stop, W1 and the switch start a small transient unit of the kit's own, t3-golive-hold-<hash of the home>, which runs t3-golive hold --home <home>. It takes the owner lock and keeps it until it is stopped. The run adopts it as its own lock. See coldcopy.hold_home and handoff.serve_holder / Holder.
  • Handover. Every start the kit makes goes through handoff.starting. That includes the update, the switch's put-back, the switch guard's unreadable-journal start, and the copy guard's start. So the new launcher gets the home from the holder, in the same way it got it from the run before. A build that takes no serving lock gets the home just before its start, as before.
  • Left holding. The holder keeps holding until T3 really runs. That covers a failure before the start; a start or update that fails, times out, or returns while the service is not active (a t3 update that exits non-zero, or a target that never comes up); the run's own cleanup (Gap.close detaches the holder and does not kill it); and a SIGKILL. It is let go only to a launcher that took the home, or once the service is active.
  • Next run. A rerun of the same back adopts the holder (holderFrom in the stop step). There is no free moment in between, and never a second holder, because each home has exactly one unit name.
  • Backstop. A resumed verify stops a holder that is still there once the service runs T3 (holderLetGo).
  • Owner tools. status and check name the holder and its unit. settle stops it while holding the journal lock, because the owner says the home is right.
  • Reboot (a known gap, documented). A reboot ends the holder. The kit finds the holder by its live process in /proc/locks (its exact argv), never by a file, so no stale state can outlive it. After a reboot, status warns that the home is free while a run that stopped T3 has not ended ok. The README gives the reasoning.
  • /proc/locks unreadable. No holder starts, because the kit could not find it again. The run holds the lock itself and says so (holderError). A holder unit that is active while /proc/locks cannot be read is named by its unit in status and check.
  • Fallback. If no transient unit can be started, the run holds the lock itself, as before. The stop step says so (holder: null, holderError).
  • Copies get no holder: a killed copy's guard starts T3 again at once.

Differs from the issue's wording: the issue says the next run should "take the lock, then stop the holder". One exclusive lock cannot be taken while the holder has it, so the next run adopts the holder instead. That has the same effect (no gap) without a free moment.

State question (tests, gap.test.py Holder, SwitchHolder)

  • Stopped half-way: a W1 that fails after its stop, and a W1 that is SIGKILLed after its stop, both leave the home owned. A fresh launch and a waited launch both wait. The same holds for a switch killed after its stop.
  • Runs again: the rerun adopts the holder. No second hold unit starts, and the home stays owned while the rerun stops T3 again. The rerun then hands the home to the launcher and ends ok with no holder left. A second settle finds no holder.
  • Old or empty state: after a "reboot" (the holder killed) there is no holder. status no longer names one, and the rerun takes a new holder. A holder still present at a resumed verify is let go to the running T3. With no transient unit available, the run falls back and says so.
  • Owner tools: status, check and settle were checked end to end. The unit's real command line (bin/t3-golive hold --home ..., with an empty environment and no machine file) was run and stopped with SIGTERM, exit 0.
  • Existing tests: W1Gap's rerun test now drops the holder with a simulated reboot before a hand server can take the home. The switch's and W1's call lists include the holder's run_detached.

Follow-ups (not in this PR)

  • A build with no serving lock ("before the start"): a failed update there leaves the home free. Fix: start a new holder when the update returns and the service is not active.
  • holderLetGo (lens A 4): restart through handoff.starting instead of only stopping the holder.
  • Watch handoverLate on the first live runs (lens B 4).
  • golive: journal the measured handover delay; load-proof gap timing test (#534) #554 minors: journal the handover in the update steps (2), systemctl on the critical path (3 / B1), the handoverLate wording (B2).
  • settle cannot stop a holder unit that it cannot see in /proc/locks. status names systemctl --user stop <unit> for that case.
  • A holder that survives a reboot (a persistent unit). It is not done because it would be on-disk state that can go stale.

General vs project

Everything here is general to the kit: upstream's owner lock (server-owner.sqlite) and its serving lock, a systemd user transient unit, and /proc/locks. Nothing is specific to one machine or project. The unit name comes from a hash of the home's real path.

Old code

Test On main's lib (back, check, coldcopy, handoff, journal, switch)
test_a_w1_that_fails_after_its_stop_leaves_the_home_owned fails: ('fresh', 'started a hand server, pid …')
test_a_killed_w1_leaves_the_home_owned_and_its_rerun_takes_the_holder_over_with_no_gap fails: a hand server starts
SwitchHolder (both) fail: home_owned is False after the killed switch
other Holder tests fail (no holder API on main)

I ran these against main's lib files with the new tests and fixture. All golive suites pass on this branch: back, check, contract, copy, gap, journal, rehearse and switch.

🤖 Generated with Claude Code

@scuttletoby

Copy link
Copy Markdown
Owner Author

Review (lens A)

Verdict: one BLOCKING finding. The fix leaves #530's own case open for a W1 whose t3 update returns without starting T3.

Checked: the full diff (handoff, coldcopy, back, switch, check, journal), the Holder/SwitchHolder tests, and upstream serviceLauncher.ts/tunnel.ts for how the holder behaves. gap.test.py passes on the branch (40 tests). With #554's lib in place of this PR's, 11 of the new tests fail or error, as the body says.

BLOCKING

  1. lib/golive/handoff.py:240: a timeout handover stops the holder even when T3 is not running. The watcher keeps the holder only when how == 'start failed', which means fn raised. Take a W1 whose t3 update returns without starting the service. Examples: it exits non-zero (the install fails, the version is not in the feed, the disk is full), or run_t3 hits UPDATE_TIMEOUT and returns code None. Then fn returns, the watcher waits HANDOVER_WAIT, sets timeout, and runs let_go(), and Holder.close() kills the holder. _update then raises, and the W1 ends with the home restored, T3 stopped and disabled, and no owner. The next SSH launch starts its own t3 serve on it, which is exactly t3-golive: a failed W1 leaves the home free for an SSH launch once the run exits #530. This hits every W1 to a build from SERVING_LOCK_SINCE on, which means every fork release from now on. The suite misses it because the Holder tests go back to fx.LIVE = 0.1.9, so the holder is let go "before the start" by design, and their failure is injected in restore, not in the update.
    • Repro (unit level): start a real holder with KIT_BIN hold --home H, then run g = Gap(H); g.adopt(Holder(H, pid)); g.starting(lambda: (3, '', 'update failed'), '0.1.11-nightly.20261118.4890', lambda: {'active': False}). The result is handover == 'timeout' and find_holder(H) is None.
    • Fix: keep the holder for timeout as well when the service is not active: if outcome.get('how', 'start failed') in ('start failed', 'timeout') and _stays(self.lock) and not _active(service). I checked that this one change makes the repro keep the holder (it survives starting() and Gap.close()). Add a Holder test that does a W1 to a serving-lock build with update.code = 3 and then calls assert_owned_after_the_run().
    • A related case that is not blocking: a W1 to a build without the serving lock lets the holder go before the start by design, so a failed update leaves the home free there too. Consider starting a new holder when fn returns and the service is not active.

MAJOR

  1. A reboot drops the holder, and that brings back the t3-golive: a failed W1 leaves the home free for an SSH launch once the run exits #530 window for a failed or killed W1 (handoff.py:43, README). "The unit is disabled, nothing starts at boot" covers the boot service, but the danger in t3-golive: a failed W1 leaves the home free for an SSH launch once the run exits #530 is the desktop app's SSH launch. After the reboot that app reconnects, finds no marker and no owner, and starts t3 serve on the half-restored home. Reboots after a failed W1 are not rare (the owner may reboot to "fix" things). Fix: make the hold survive a reboot. For example, the W1 stop step could write a small persistent user unit (t3-golive-hold@<hash>.service, WantedBy=default.target, enabled) with the same command line, removed by the run that hands the home to T3 and by settle. Alternatively, document the reboot case as a known gap in the README's state table, and have status say it after a reboot (journal shows W1 stopped T3, there is no holder, and the service is disabled).

MINOR

  1. coldcopy.py:690 and handoff.py:452: when /proc/locks cannot be read, hold_home still starts the holder and can then never see it. find_holder returns None, _guard_alive stays true, and the loop runs the whole STOP_WAIT (120 s). Then hold_owner_lock(..., 120) times out against the holder, and the stop step fails after about 4 minutes, leaving a holder that status, check and settle cannot find, so it holds until a reboot. The docstring says this case falls back to the run's own lock. Fix: check handoff.lock_holders(owner) is None before run_detached, and in that case take the lock in-process right away.
  2. back.py:431 (holderLetGo) frees the home while T3's launcher is idle. The service is "active" while a holder holds only because upstream's launcher met the owner lock, released its serving lock and fell into #idleWhileStateDirOwned. It polls every 30 s and needs 2 free probes, so it comes back 30–60 s later. Stopping the holder there leaves the home unowned for that time, and an SSH launch in its home-owned loop (1 s polls) takes it. The path is rare (it is a backstop), but the fix is cheap: restart the service through handoff.starting(home, adapter.restart, version, adapter.service) with the holder adopted, so a fresh launcher claims the serving lock before the holder goes.
  3. switch.py:770: the switch guard's "journal unreadable" start now also hands off the holder. The fail-open start was already there before this PR. But the guard does not check svc['enabled'], unlike the copy guard. If the switch's own automatic W1 disabled T3 and was killed mid-restore, this path starts T3 on a half-restored home and removes the holder that was protecting it. Suggest skipping the start (and the hand-off) when the service is disabled, as coldcopy.guard does.

State question:

  • Runs again: the rerun adopts the holder (one unit name per home, --collect so a failed or killed unit does not block the name), with no gap.
  • Stops half-way: SIGKILL after the stop is fine. After the update with a no-serving-lock build it is fine. During a timeout it is not (finding 1).
  • Existing or old state: a holder from before is adopted, and check refuses while one exists. A reboot leaves no holder (finding 2).
  • Holder left when nobody will release it: only via finding 3, or if the owner starts T3 by hand (the launcher idles, and status names the holder and settle stops it, which is acceptable).

@scuttletoby

Copy link
Copy Markdown
Owner Author

Review (lens B)

Head 5c7cde0. Read #530, the diff and the body; ran all golive suites on this head (all pass). Reverted the fix (W1's stop step back to hold_owner_lock in the run's own process): Holder.test_a_w1_that_fails_after_its_stop_leaves_the_home_owned and ..._killed_w1_... both fail as the body says. So the tests do test the fix, but they cover only a failure in the restore step.

BLOCKING

1. A W1 whose t3 update fails stops the holder and leaves the home free. lib/golive/handoff.py:233-243, with lib/golive/back.py:413-419.
W1's update step calls handoff.starting(home, lambda: live.run_t3(argv, ...)). run_t3 does not raise when t3 update exits non-zero or times out: it returns (code, out, err). So fn "returned", the watcher waits HANDOVER_WAIT, ends with 'timeout', and the finally calls let_go(). That is Holder.close(), which SIGTERMs the holder. Only after that does _update raise exited 3. The holder is kept only for 'start failed', and this path never reaches it.
Failure: W1 → stop → holder → restore ok → t3 update exits 1 (or hits UPDATE_TIMEOUT) → about 20 s later the holder is gone. W1 then ends failed with T3 stopped and disabled, and nothing owns the home. The next SSH launch starts its own t3 serve on it. That is exactly the case #530 is about ("a W1 that fails after its stop").
Reproduced on this head with the PR's own fixture: a Holder subclass test that writes update.code = 3, runs back('w1'), then calls assert_owned_after_the_run(). Result: service active: False, enabled: False, find_holder → None, and the race gives ('fresh', 'started a hand server, pid …').
The same thing happens in the switch when t3 update exits 0 but the target never comes up (crash loop, killed in a migration). That ends in 'timeout' too, so the home is free while the switch is still in its verify, before its W1 takes a new holder.
Fix: never let a holder go unless T3 is up. In the watcher's finally, use the "keep it" branch for every outcome other than 'launcher' when _stays(self.lock) and not _active(service), not only for 'start failed'. With the service active, timeout can still let go as today. Add the test above (update exits non-zero, and update returns 0 with the service never active), and add it to the state table.

MAJOR

2. With /proc/locks unreadable, the holder still starts, then can never be found, adopted or stopped. lib/golive/coldcopy.py:684-712, lib/golive/handoff.py:447.
find_holder returns None when lock_holders is None. Its docstring says "(the kit then never starts one: it could not tell it holds)", but hold_home calls run_detached without checking. What follows:

  • The holder unit takes the lock and stays alive, so the loop waits the full STOP_WAIT (120 s).
  • hold_owner_lock then times out against the holder (another 120 s), find_holder is still None, and TimeoutError is raised. The stop step fails with T3 stopped.
  • The holder now keeps the home owned, and nothing in the kit can see it: status and check do not name it, and settle does not stop it.
  • A rerun's run_detached fails ("unit exists"), and its fallback times out the same way. The home stays locked until someone stops the unit by hand or reboots.
    This depends on the environment (a container or a hardened /proc), but it is a lock that nobody will release.
    Fix: in hold_home, if handoff.lock_holders(owner) is None, do not start a holder. Fall back at once to the in-process lock with holderError, as the docstring says.

MINOR

  1. A manual start while a holder exists leaves T3 idle with no time limit. systemctl --user start|restart t3code.service while a holder exists gives a launcher whose server meets the holder's lock (250 ms) and idles, and the holder never yields. Before this PR, the lock died with the run. holder_said could add: "do not start T3 by hand while it runs; run the same back or settle." status already names the holder, so this is only a wording gap.
  2. The handover to a holder costs more than a local close. Each handover now includes a SIGTERM to another process, its wake-up and exit, and 2 ms polls of /proc/locks. handoverMs (golive: journal the measured handover delay; load-proof gap timing test (#534) #554) measures it, which is good. Under heavy load this eats into the server's 250 ms more than the old in-process close did. No change needed now, but watch handoverLate on the first live runs.

State question, the rest checked and fine: a SIGKILL anywhere after run_detached leaves the holder, and the rerun adopts it with no second unit (one name per home). A reboot drops it and nothing stale is left (found by argv in /proc/locks, not by a file). settle runs under the journal lock. The switch's guard and the copy guard start T3 through handoff.starting, so they take a holder over. A W1 killed mid-restore is never let go by the switch guard (W1_HOLDS). All T3 starts in the kit go through handoff.starting. Merges cleanly with #557.

Verdict: NOT SAFE until 1 is fixed (2 should go with it).

scuttletoby added a commit that referenced this pull request Nov 20, 2026
…cks is blind (#530)

Lens round on #556:
- The handover keeps the holder for every outcome but the launcher's claim while the service is not active: a
  `t3 update` that exits non-zero or times out, and a target that never comes up, no longer free the home.
- hold_home starts no holder when /proc/locks cannot be read (it could never be found, adopted or stopped); the
  run holds the home itself and journals why. status and check name an active holder unit they cannot see there.
- A reboot ending the holder is a known gap, documented; status now says the home is free then.
- The switch guard's fail-open start leaves a disabled T3 (its W1 holds it) stopped and held.
- holder_said: do not start T3 by hand while it holds.
- #554's look-order test stamps the claim before BEGIN EXCLUSIVE (no GIL race); `looked` is set before the
  watcher starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby
scuttletoby force-pushed the fix/golive-530-holder branch from 5c7cde0 to 1703f80 Compare November 20, 2026 00:33
@scuttletoby
scuttletoby changed the base branch from fix/golive-534-handover-delay to main November 20, 2026 00:33
scuttletoby and others added 3 commits November 20, 2026 00:47
…530)

A W1 (and a switch) may fail or be killed with T3 stopped on purpose (the home may be half-restored). The owner
lock it took after the stop was held by the run's own process, so it was gone with the run: the next SSH launch
of the desktop app found the home free and started its own server on it.

Right after the stop the kit now starts a holder, a transient unit of its own (t3-golive-hold-<hash of the home>,
running `t3-golive hold --home <home>`), that takes the owner lock and keeps it until stopped; the run adopts it
as its lock. Every start the kit makes goes through the handover, so the new launcher gets the home from the
holder. A failure before the start, a failed start or a kill leaves the holder holding. The same back's rerun
takes it over (no free moment, one unit name per home); a resumed verify lets a leftover holder go once the
service runs T3; status and check name it; settle stops it; a reboot ends it. It is found by its live process in
/proc/locks, never by a file. Where no transient unit starts, the run holds the lock itself, as before, and says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cks is blind (#530)

Lens round on #556:
- The handover keeps the holder for every outcome but the launcher's claim while the service is not active: a
  `t3 update` that exits non-zero or times out, and a target that never comes up, no longer free the home.
- hold_home starts no holder when /proc/locks cannot be read (it could never be found, adopted or stopped); the
  run holds the home itself and journals why. status and check name an active holder unit they cannot see there.
- A reboot ending the holder is a known gap, documented; status now says the home is free then.
- The switch guard's fail-open start leaves a disabled T3 (its W1 holds it) stopped and held.
- holder_said: do not start T3 by hand while it holds.
- #554's look-order test stamps the claim before BEGIN EXCLUSIVE (no GIL race); `looked` is set before the
  watcher starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@scuttletoby
scuttletoby force-pushed the fix/golive-530-holder branch from 1703f80 to 819033c Compare November 20, 2026 00:51
@scuttletoby

Copy link
Copy Markdown
Owner Author

Fix round

Head 819033c52e. Rebased onto main twice: first after #554 merged (--onto origin/main 24c105f), then after main was moved on 11-19 (--onto origin/main f49745f). The patch-ids match, so only this PR's commits came over. The base is now main.

Item Change Test On the old code
BLOCKING (A+B): the holder is let go when t3 update exits non-zero or times out, or when the target never comes up The finally in Gap.starting keeps a holder unless a launcher took the home or the service is active. A crash loop shows activating, so it stays held Holder.test_a_w1_whose_update_returns_without_t3_up_leaves_the_home_owned (update exits 3; the rerun adopts); test_a_w1_whose_build_never_comes_up_keeps_the_home_owned_through_its_reruns (2 failed runs); SwitchHolder.test_a_switch_whose_target_never_comes_up_keeps_the_home_owned_until_its_guard_puts_t3_back (killed after update, then the guard) all 3 fail: the holder is None, home free
MAJOR (B): /proc/locks unreadable coldcopy.hold_home starts no holder when the owner lock's holders cannot be read. The run holds the lock itself and says so (holderError). New handoff.holder_seen: status and check name an active holder unit by its unit name when /proc/locks is blind test_with_proc_locks_unreadable_no_holder_starts_and_a_rerun_still_ends_ok; test_a_holder_unit_status_cannot_see_in_proc_locks_is_named_by_its_unit first errors (TimeoutError: the rerun cannot take the home); second fails (the unit is not named)
MAJOR (A): a reboot drops the holder Documented as a known gap, not fixed. The unit is transient and disabled, and nothing starts at boot. A persistent holder would be on-disk state that can go stale, which this design avoids. Instead, status now warns (back.NO_HOLDER) when there is no holder, T3 is stopped and disabled, and a run that stopped it has not ended ok. The README has a "Known gap" bullet test_after_a_reboot… asserts the warning, that the rerun works, and that the warning is gone afterwards errors: AttributeError: NO_HOLDER
Related: the switch guard with an unreadable journal started a disabled T3 It leaves a disabled, stopped T3 stopped (a W1 that stopped it may still hold it) test_a_guard_that_cannot_read_the_journal_leaves_a_disabled_t3_stopped_and_held fails
#554 minor (A): flaky timing race in its new test claimed is stamped before BEGIN EXCLUSIVE, and looked is set before the watcher starts Shown by adding a 0.1 s sleep between BEGIN and the stamp: old order [] != [True], new order passes fails with the sleep
One-liners holder_said adds "do not start t3code.service by hand while it holds" covered above n/a

The rest are listed as follow-ups in the PR body.

State question: reruns adopt the holder; there is never a second unit. A run killed half-way leaves the home held. A reboot (old state) leaves no holder and status warns. With /proc/locks blind, no holder starts and nothing is left that cannot be found.

Suites on this head, all OK: gap 46, back 106, check 71, contract 5, copy 108, journal 29, rehearse 87, switch 101. To get the old-code results, I ran the new tests and fixture against this branch's lib from before the fix commit.

@scuttletoby
scuttletoby merged commit c174db9 into main Nov 20, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

t3-golive: a failed W1 leaves the home free for an SSH launch once the run exits

1 participant